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ABSTPACT 

A number of issues related to the development of 
evaluation methodology are discussed. These issues are as follows: 
the definition of the purpose, general strategies for the development 
of methodology, and methodological research. The evaluation 
methodology developed is for the purpose of providing data for 
decision making. Some specific implications for the use of the 
evaluation methodology are: the goals evaluated should l>e the 
decision maker's goals for the enterprise; the variables measured 
should be those of concern to the decision maker; and the 
observational techniques used should possess decision-maker validity* 
(Author/CK) 
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INTRODUCTION 

Some evaluation methodology has been p *oduced for the purpose of providing 
data for decision making. Unfortunately no; all of this methodology has attended 
to some important implications of this purpose. This paper attempts to Identify 
these overlooked implications and discuss tl.eir impact on the development of 
evaluation methodology where the word methodology is defined as a systematic, 
operationalized, standardized set of rules tnd procedures designed to accjeplish 
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The j>aper deals with a number of issue; related to the development o: 
methodology. These issues are as follows: the definition of the purpose, 

general strategies for the development of methodology, and methodological 
research. These topics will be considered in the context of the development of 
evaluation methodology. 

Toward a Definition of the Purpose 

If the purpose of evaluation is to provide data for decision making the 
implication is that in a successful evaluation the data is actually used in the 
decision making process. If the data is actually used there is the implication 
that it is used by decision makers, i.e. real people who used the data produced 
by the evaluation for their decision making purposes. There is a further impli- 
cation that successful evaluation methodology produces evaluation designs that 
produce data that is actually used for decision making by the decision makers 
for whom the data was collected. 
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If a datum produced by the evaluation is used by a decision maker 
say that the datum has decision maker validity for that decision maker, if a 
decision maker does not use a datum presented to him then the datum lacks de- 
cision maker validity. Evaluation methodology should attempt to maximize de- 
cision maker validity in evaluation designs and the data produced. 

Perfect (100Z) efficiency in an evaluation design mould exist vhere 
for every decision maker for whom data mere provided every datum mas used by 
that decision Baker in his decision making. Zero efficiency mould exist *here 
no datum was used by any decision maker. Thus, if the use of data for de is ion 
amtking were observable the efficiency of an evaluation design could be quantified 
by observing t le percentage of the data presented that mas act ally used T y 
decision Baker?. 

Development of Methodology 

Given a defined purpose it becomes possible to develop systematic, st an— 
dardized, operationalized rules and procedures for the accomplishment of the 
purpose. The definition of the purpose provides for the logical testing of altem 
tive procedures. For example, if the evaluator provides data to a decision maker 
vhere the data are not relevant to the decision maker's intents for that enter- 
prise, then the decision maker will consider the data irrelevant and he will not 
use the data in his decision making processes. The purpose of the evaluation will 
have failed and the resources spent on the evaluation will have been wasted. 

Once rules and procedures for an evaluation methodlogy have been developed 
to the point where no lcgical flaws can be identified it becomes reasonable to 
perform research on the methodology. Methodological research is also made pos- 
sible by the existence of defined criteria for success. The most parsemonious 
first methodological research is a decision oriented field test. 



In a field test the methodology is implemented in a single sitnatior. 
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• Observations are Bade on whether the steps of the Methodology can be performed. 
Plullj i the percent of data that was actually used in the decision suiting process 
by the persons for whoa the data was- developed is observed. If a step i.i the 
Methodology tails to work it can be redesigned and retested. If the methodo- 
logy is totally successful then it cannot »<e said that it would be successful 
for other decision Bakers or other enterprises. 

When dec is ion oriented nethodological research no longer uncovers a »thodo- 
logical problems the researcher aoves to conclusion oriented nethodologi :al re- 
search. He has two options. He can replicate the application of the net todology 
over a diversified set of enterprises and decision makers. If it is suc:essful 
every tine it can be considered conplete. If at any point it fails, the 
methodology cm be redesigned to avoid future failure. The other option is to 
draw a random sample of decision makers and enterprises and use infarent al 
statistical approaches. 

The existence of a defined purpose is the key to methodological development. 

It provides the criteria for logically testing alternative steps in the 
methodology. It provides the criteria for a field test of the methodology. 

It provides the criteria for conclusion oriented methodological research. A 
defined purpose makes possible the systematic development of a set of rules and 
procedures to accomplish that purpose. 

Completeness of Data Provided 

There may be other criteria for evaluation methodology in addition to the 
percent of data actually used by the decision makers for whom the data were 
collected. If a decision maker for whoa data are collected makes decisions 
where no data were used to assist in his decision making process then the 
evaluation is incomplete. The evaluation is incomplete to the extent that de- 
cisions are aade without the help of data. If we can know what decisions are 
made with and without data then the percent of decisions made without data is 

3 



- 4 - 

* a quantification of the incompleteness of the evaluation. 

However, the resources available for evaluation becone a possible constraint 
upon coupleteness • The resources for evaluation are generally liaited tc the 
point where 100 percent coupleteness is inpossible. In this nost likely situa- 
tion the maxinna coupleteness within the resource constraint is desired. 
Furthermore, the evaluation should distribute the completeness such that data 
is provided f>r the decision maker's more iaportant decisions and not pro- 
vided for the least important decisions. If we could know the decision raker's 
decisions prinritorized by iaportance then the correlation between the pr iorities 
and whether o** not data was provided would he a quantification of the focus of 
the completeness. This criteria also suggests that the decision maker's 
priorities should be built into the evaluation design so that the highest pos- 
sible degree of focus can be assured. The use of the decision maker's priorities 

should be bui. t into the operationalized st?ps of evaluation methodology. 

- • 

Decision Maker Validity Within Steps of Met rodology 

The criteria of efficiency, the percen: of data used by the decision maker 
is a very useful criteria for identifying the essential steps of an evaluation 
methodology designed to provide data for decision malting. It suggests that che 
methodology should ensure at every step that decision maker validity is pre- 
served in the evaluation design. 

Every person who will be provided with the data produced by the evaluation 
should be treated as a decision maker and concern for decision maker validity 
should govern what data should be presented to each person. If the report to 
the decision maker has ten useful data items hidden among 1,000 non-useful data 
items there is a considerable chance that the decision maker will not be willing 
to dig out the data that he would otherwise find useful. 

The person who has legal control of the evaluation resources at the time of 
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hiring the evaluator should determine which decision makers should be 
presented with data developed for their decision making and the priorities 
analog them such that the completeness across derision makers is allocated ac- 
cording to the desirability of their receiving data. The evaluator should 
assist by presenting alternatives such that a decision to not provide a parson 
with data is a deliberate decision rather than oversight. 

If decision maker validity is to be preserved the data collected foi a 
particular decision maker must be perceived by that decision maker to be 
relevant to his intents or goals for that enterprise. That is, the data mist 
help him to mcke decisions so that he can try to cause the enterprise to 
accomplish the things he really wants the enterprise to accomplish for hiiself 
and others. Cifferent decision makers will have different intents for th? 
same enterprise. The listed project goals rill most likely fail to inciule scce 
of any particular decision maker's goals (i.e. be incomplete) and nay include 
wae goals that he does not hold (i.e. be it efficient). This is particul nrly 
an issue when the decision maker had no part in preparing the project proposal. 

The enterprise should be evaluated in terms of its parts so that the 
decision maker can identify the troublesome areas. Further, these parts rust 
be meaningful to the decision maker. If the evaluator reports data on the 
school/society interface and this concept has no meaning for the decision 
maker, then he will have a hard tine using the data. 

The specific evaluative variables must be the decision maker's opera- 
tionalizations of the vague terms in his goals. Not everyone wanting students 
to be "good citizens" means the same thing. The goal "individualized instruction" 
has, perhaps, more different meaning. And "positive self concept" may veil have a 
different meaning for every person who uses the term. If we wish to maximize 
decision maker validity through the efficiency of the evaluation design then 
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* evaluation methodology mist ensure that ve deal with the decision maker's 
meaning in the selection of specific evaluative variables* 

Decision maker validity may be thwarted through inappropriate observational 
techniques, the observational technique should be carefully explained to the 
decision naker* When he says n I don’t trurt a teacher's log for that ki id of 
data," use a different technique* If the decision naker is unaware of th» ob- 
servational techniques used it makes it eas/ for him to ignore data that a ay 
make him feel Uncomfortable. It is very ha d for a decision naker to ignore 
data he has said that he would use when it is collected in a manner that ie has 
approved. The approval of the decision maker must be a commitment and no: 
merely a verbalisation. 

Assuming that the evaluation desigr. has not lost decision maker valiiity 

through any of the problems described above the evaluator's last chance t> 

confuse the decision maker is in data analysis. If the decision maker is un- 
« 

sophisticated in multivariate analysis (and most decision makers are un- 
sophisticated) he can report the data in terns of eigenvalues, percent 
of trace, discriminant vectors, and Beta weights. If the evaluator interprets ^ 
the meaning of a alyses then the data presented to the decision naker is the 
evaluator's interpretation. This kind of data frequently lacks decision uaker 
validity although this nay not be obvious to the evaluator since the decision 
maker may wish to spare his feelings. 

Sometimes the evaluator will explain a statistical technique to the decision 
maker, e.g. a thirty minute short course in multiple regression. Teaching an 
analysis is a solution to this problem only when the necessary understanding is 
genuinely developed prior to deciding upon that form of analysis. 
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SUMMARY 

Sove evaluation methodology has been developed for the purpose of pro- 
viding data for decision Baking, Unfortunately not all of this methodology 
has attended to seme important implications of t.iis purpose. The purpose is 
fulfilled if, and only if, the data provided is actually used in the decision 
making process. To the extent that (a) the evaluation produces data that is 
not used, that evaluation is wasteful; and T>) the evaluation produces no data 
for some decisions that needed to be made, : t is incomplete. 

More Importantly, however, it is necessary to take account of the fa't 
that decision Bakers are real people. If the person or persons for whom :he 
data is collected do not believe in the utility of the data provided they will 
ignore that data, the evaluation resources used to generate that data mil.. have 
been wasted, and the purpose, for the evaluation will have failed. An eva uation 
methodology th it does not assure that the data will be used does not accottplish 
the purpose: to provide data for decision raking. 

Some specific implications of the above for evaluation methodology are 
os follows : the goals evaluated should be the decision maker's goals for the . 

enterprise, the variables measured should be those of concern to the decision 
maker, the observational techniques that are used should possess decision 
maker validity: the enterprise should be evaluated in terms of its parts as 

conceptualized by the decision maker, any data analysis performed should be 
comprehensible to the decision maker by the time that analysis is presented 
to him. 



